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Abstract 

Discrete flavor symmetry is explored for an intrinsic property of mass matrix forms 
of quarks and leptons. In this paper we investigate the permutation symmetry 
and derive the general forms of mass matrices in various types of 6*3 theories. We 
also exhibit particular realizations of previous ansatze of mass matrices, which have 
often been applied in the literature to the standard model Yukawa sector. Discrete 
flavor symmetry is also advantageous for vanishing matrix elements being dynam- 
ically generated in the vacuum of scalar potential. This is due to the fact that 
group operations are discrete. While zero elements themselves do not explain mass 
hierarchies, we introduce an abelian flavor symmetry. A non-trivial issue is whether 
successful quantum numbers can be assigned so that they are compatible with other 
(non-abelian) flavor symmetries. We show typical examples of charge assignments 
which not only produce hierarchical orders of mass eigenvalues but also prohibit 
non-renormalizable operators which disturb the hierarchies in first-order estima- 
tion. As an explicit application, a flavor model is constructed in grand uniflcation 
scheme with S-i and U{1) (or Z^) flavor symmetries. 



''On leave of absence from Institute of Theoretical Physics, University of Tokushima, 770-8502, Japan. 



1 Introduction 



One of the most important issues in and beyond the standard model is the masses and 
mixing angles of the three-family quarks and leptons. After the electroweak symme- 
try breaking, the observed values of masses and mixing angles are to be produced from 
the structures of Yukawa couplings. However the Yukawa couplings generally have re- 
dundancy in explaining the experimental results: apparently different forms of Yukawa 
matrices lead to the same physical masses and mixing angles. Therefore it has been a long 
outstanding problem which patterns of Yukawa couplings are relevant from phenomeno- 
logical and theoretical viewpoints. Various progresses have been made in the literature by 
applying additional principles to the standard-model Yukawa sector. The two well-known 
examples of such principles are to adopt the unification hypothesis of matter multiplets 
and to assume specific forms of Yukawa couplings ( "textures" ) . The former is a top-down 
approach to the problem. The grand unification principle relates the properties of quarks 
and leptons, and reduces the degrees of freedom of Yukawa couplings in the theory. On 
the other hand, the latter approach is rather a bottom-up one. Available forms of Yukawa 
textures are explored so that they are consistent with the experimental observations. It is 
interesting that the number of successful textures is found to be highly limited. This fact 
is revealed from a simplifying assumption that some elements of Yukawa matrices vanish, 
called texture zeros. Along this line, phenomenologically possible forms of mass matrices 
have been proposed in the literature (for example, [H 12 El , and also jH |3] for systematic 
analyses of zero textures). 

As for the neutrino sector, the recent experimental results suggest that one of the most 
likely forms of Majorana mass textures of light neutrinos is proportional to 



in the basis where the generation mixing has been rotated out in the charged-lepton side. 
This is the dominant part of mass matrix and other small entries have not explicitly 
been written down. It is clearly seen that the large leptonic mixing between the second 
and third generations is predicted from The observed large 1-2 mixing re- 

quires an additional condition for the above texture: the dominant 2x2 sub-matrix of 
fjl.l|) has a reduced rank and its determinant is of the same order of small off-diagonal 
elements neglected in Given that, the texture form (jl.ip is considered as a promis- 

ing candidate consistent to the present experimental data. The condition (the vanishing 
determinant of the dominant sub-matrix) may be realized without fine tuning of model 
parameters, e.g., with the right-handed neutrino dominance i?-parity violation 
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and the lopsided form of charged- lepton mass matrix ^UJ- It is interesting that the lop- 
sided mass textures can be naturally embedded in grand unified theory. However if the 
theory is supersymmetrized, large off-diagonal elements in the lepton Yukawa matrices 
generally induce sizable rates of flavor- violating processes to excess the present experi- 
mental bounds jTTj. While the details depend on superparticle mass spectrum, a natural 
way to avoid this flavor problem is to consider the case that the lepton as well as quark 
Yukawa matrices take hierarchical forms, which lead to small generation mixing. If this is 
the case, an interesting possibility to have large lepton mixing is to suppose asymmetric 
forms of mass textures (zeros).* It seems that abelian flavor symmetry may be difficult 
to generate such asymmetric zeros without expense of model complexity. Furthermore 
non-abelian continuous flavor symmetry is not suitable for handling mass textures since 
texture zeros are rotated to other arbitrary forms by continuous symmetry rotations and 
do not have physical implications. 

Motivated by these results, in this paper we investigate the power of non-abelian dis- 
crete flavor symmetry for constructing mass matrix models. We particularly focus on the 
minimal discrete non-abelian symmetry 5*3. (For fermion mass models based on other 
non-minimal discrete non-abelian flavor symmetries, see JH]-) The 5*3 operation is the 
permutation of three objects, which has a simple geometrical interpretation, i.e. the sym- 
metry of an equilateral triangle. While it is the smallest non-abelian discrete symmetry, it 
might be regarded as a remnant of flavor symmetry of fundamental theory in high-energy 
regime. The purposes of this paper are the following two points: (i) non-abelian flavor 
symmetries such as 5*3 are so effective that various types of phenomenological textures are 
handled and (ii) these symmetries can also generate asymmetric forms of Yukawa matri- 
ces in dynamical ways. Further it should be noted that texture zeros themselves do not 
explain fermion mass hierarchies. Previous approaches to fermion masses with 5*3 have 
assumed hierarchical values of Yukawa couplings and/or involved symmetry-breaking pat- 
terns such as vacuum expectation values (VEVs) of Higgs scalars. In this paper we show 
that realistic values of masses and mixing angles are dynamically achieved by introducing 
f/(l) symmetry. A non-trivial problem arises whether successful U{1) quantum numbers 
can be assigned so that they are compatible with other non-abelian (flavor) symmetries. 
It is noticed that mass hierarchy is also realized in a similar way with a discrete subgroup 
of the flavor U{1) symmetry such as Z^- with appropriate (enough large) and the same 
quantum numbers as in the U{1) case. A smaller choice of A^ would be possible and 
interesting from a viewpoint of brevity. In this case, the problem of fermion masses can 
be handled with flavor symmetries that are entirely discrete. While the U{1) charge as- 

*Leptonic mixing angles may be enhanced, e.g. by integrating out heavy fields such as right-handed 
neutrinos [T^ . 
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signments are presented in this paper, they can always be read as the charges in flavor 
theory. We flnally present an exphcit model where the 5*3 flavor symmetry is incorporated 
consistently to unified gauge symmetry and hierarchical forms of mass matrices. 

This paper is structured as follows. In the next section, we discuss some fundamental 
issues of the S3 group, which are needed in Section 3 to study symmetry-invariant forms 
of mass matrices. In Section 4, 5*3 is applied to supply dynamical justifications to realistic 
candidates of mass textures which have often been discussed in the literature. Based 
on these results, we present in Section 5 a toy fiavor model in grand unification scheme 
where Yukawa textures are controlled by a single fiavor 5*3, assisted by U{1) symmetry. 
In Section 6, we analyze the invariant scalar potentials of S3 doublet whose VEV form 
is a key ingredient of the approach developed in this paper. Section 7 is devoted to 
summarizing our results. 



2 The S3 group 

2.1 Representations and representation matrices 

The 5*3 symmetry is the smallest non-abelian symmetry, the permutations which an equi- 
lateral triangle has. The S3 group therefore contains six elements Ti, Tg, half of 
which are the circulations of triangle apices and the other half corresponds to the ex- 
changes of two of three apices while the other is fixed. There are only few numbers 
of irreducible representations; a two-dimensional representation and two different one- 
dimensional representations. Throughout of this paper we denote them as 2 (doublet). 
Is (singlet), and 1a (pseudo singlet), respectively. The non-trivial one- dimensional rep- 
resentation 1a distinguishes the group elements in two parts. The matrix representations 
of group elements are given in Tabled It is easily seen that the even permutations Ti, 
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Table 1: The representation matrices of the 5*3 elements. The symbol x is the third root 
of unity (x = e^''*/^). 

T2 and T3 constitute the subgroup Z3. This means that the 5*3 symmetry is broken down 
to Z3 when a pseudo-singlet field 1a develops an expectation value. Thus the 5*3 group 
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has simple but non-trivial structures, and is suitable for applying it to the flavor prob- 
lems of three-generation fermions in the standard model. A number of models have been 
proposed to explain Yukawa coupling structures of quarks and leptons with the 5*3 flavor 
symmetry [TH IT^ ITB] . 

It may be convenient to introduce the reducible three-dimensional representations for 
discussing the three-generation flavor physics. Corresponding to Is and 1a, there are two 
types of three-dimensional representations: 

3s = 2 + Is, (triplet) (2.1) 
3a = 2 1a- (pseudo triplet) (2.2) 

If 5*3 is regarded as a remnant of some gauge symmetry in fundamental theory, 3a should 
be apphed not to induce discrete gauge anomaly. This is understood from the fact that the 
three-dimensional vector representation of 5*0(3) is decomposed as 3 = 2 -|- 1a in terms of 
its subgroup 5*3. One may also use the 3s representation at the expense that 5*3 is assumed 
to be a global symmetry or the anomaly is cancelled by introducing appropriate numbers 
of pseudo singlet fermions. Further the aforementioned geometrical interpretation of 
operations is made clear for triplet representations. Such an interpretation is seen in a 
different basis of 5*3, as will be discussed later in this section. The matrix representations 
of 3s and 3a are read from Table Q and given by 




for the 3s representation. As for 3a, the matrices Tj are given by ()2.3|) except that the 
3-3 elements in the odd permutation matrices T4 5 6 are replaced with —1. 

2.2 Tensor products and 2* representation 

The tensor products involving 1a are given by 1a x 1a = Is and 1a x 2 = 2. The only 
remaining non-trivial product is that of two doublets: 2 x 2 = 2 + 1a + Is- In the basis 
where the group elements are given by Tabled the product of two doublets il) = {ipi, 1^2)^ 
and (f) = ((/>!, ^2)* is explicitly written as follows: 

^x0 = (V^V0, V^V_0)*2 + (V^W)iA + (V^V)is, (2-4) 

= {4<P2, 4<Pl)l + {4<Pl - 4<P2)U + (^iVl + 4<p2)ls, (2-5) 
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where (Tj {i = 1,2,3) are the Pauh matrices and cr± = 2i±i£2^ 'jj-^g subscripts in the 
right-handed sides denote S3 representations. Note that ip (and also 0) is generically 
complex-valued, while S3 is a real group. This fact is important when transforms as 
some complex representation under other symmetries than 53. The complex conjugate 
ip* belongs to the 2* representation for which the representation matrices become T*. In 
practical use, however, it is convenient to define a doublet from an anti-doublet: 

= ^ir = fSl , (2.6) 



which transforms as ip^^ — ^ (TilTiip)* = TiCTiip* = Tiip^, and indeed acts as the 2 represen- 
tation of 5*3. The tensor product involving tp^ is hence given by 

^^x0 = #a^j0)* + (^*za20)u + (^Vi0)i3, (2.7) 

= (V^202, V^10i)2+(^102-V'20i)ia + (^102 + ^20i)is, (2.8) 

where ai = and = ^-y^, respectively. It is noticed that this product does not 
contain any complex conjugates. Such a type of tensor product is necessary for describing, 
e.g., matter interaction terms {F terms) in supersymmetric theory and Majorana masses 
for neutrinos. On the other hand, the product ()2.4|) is applied to the usual Dirac mass 
terms of quarks and leptons. The form of tensor product depends on the group basis, and 
therefore the 5*3 model construction and its physical consequences also do. For details, 
see the next subsection and Section IHl 

It may be useful for later discussion to explicitly write down what types of singlet com- 
ponents are contained in the products of more than two doublets. In the basis discussed 
here, one finds 

^ X D + iPl(f)2, V'102 + V'201, (2.9) 

1p X (j)X ip D 1pl4>2V2 + i^l4>lVl, 1pl4>2'^l + ^20IV52, 

ijlCpl^pl + ^202V5l, i^l4>l<^l + ^202V52, (2.10) 
ijj X (p X if X X D i'lcplviXl + V^|02V52X2, -iplCpWlXl + V'202V^2X2, 

^I^IV^IXI + V^202V52XL 1p\(plVlX2 + 1pl(p2'f2XU 

i'l(p\VlX2 + i'2(p\v2Xu MiAx2 + '^p2(p2'Axi, 

1pl(p2<f2xl + ^2(pl^lxL i'lh^lXl + 1pl(plV2X2, 
i'\(plV2Xl + i'l(p2VlX2, ^l(plV2X2 + 1p2(p2VlXl, 
i^l(p2<^lX2 + i^2(pl<^2Xl, i'l(p2V2Xl + i'2(plflX2- (2-11) 

Their hermitian conjugates are also in the Is representations. 
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2.3 Group basis dependence 

We have presented the 5*3 algebra in the complex basis where the representation matrices 
are given by those in Table H There are, however, several bases of the S3 matrices 
which are often used in the literature. So it may be instructive here to describe the 
relation among these group bases, compared to the complex basis used in the previous 
subsections. 

2.3.1 Democratic basis 

The democratic basis is adopted to produce a flavor-democratic mass matrix in which all 
matrix elements are equal jT3] . In this basis, the operations generate the permutations 
of three objects, for example, the exchange of the first and second indices. The invariance 
under such transformations require the universal size of couplings for three generations 
if they belong to a three-dimensional representation of S3 and the Higgs field is in the 
singlet. 

Unlike in the complex basis, the representation matrices have apparently non-trivial 
(not block-diagonal) structure for three-dimensional representations. Different group 
bases are converted to each other by unitary transformations. It is worth noting that 
there are two types of democratic basis which correspond to the existence of two three- 
dimensional representations; 3s and Sa- The elements in the democratic basis for 3s are 
defined by the unitary matrix V as 

Tf = VTiV\ (2.12) 
where Tj's in the right-handed side are given in the complex basis, and V = UPs with 

/ I/V2 1/V6 1/V3\ /I/V2 -I/V2 \ 

U = -I/V2 1/VQ l/VS , Ps = [ VV2 z/V2 . (2.13) 

V -2/Vq i/VsJ \ l) 

Thus the representation matrices for 3s are given by the label-changing matrices: 




(2.14) 



For the pseudo-triplet representation 3a, the matrices 7i^2,3 have the same forms as above, 
but T/gg become rather complicated, as can be seen by the definition ()2.12|) . 



6 



Another democratic basis is defined so that the representation matrices for 3a are 
expressed by label-exchanging matrices like ()2.14|1 . The matrices are given by a 
similar unitary rotation to ()2.12|) . except that the matrix P5 is now replaced by 

Pa = \t/V2 -VV2 1 . (2.15) 

One can see that the matrices Tj^2 3 ^^^^ ^^e same forms as those in ()2.14|) both for 
triplet and pseudo-triplet representations. The differences appear for T45 6; 2^4^5,6 (3a) = 
— T^5g(3s) and T^5g(3s) = — T^5g(3A), the former matrices exchanges the three objects 
as ()2.14|) and the latter have some complicated forms. 

2.3.2 Real basis 

In the discussion of the democratic basis, Ps^a rotate only the first and second indices. 

S A 

This means that the representation matrices ' are rotated to block- diagonal forms only 
by the U rotation, namely, the [/-rotated matrices are decomposed into the matrices for 
irreducible representations. (Note that U is the unitary rotation which diagonalizes the 
so-called flavor-democratic mass matrix.) The real basis is defined by rotating T^'^ 
with the U matrix, for example, 

rpu ^ u^TfU. (2.16) 
These group elements take the following forms for the 3s representation: 

'1 \ / -1/2 V3/2 \ f-l/2 -v^/2 

= I 1 I , T!^ =\ -v^/2 -1/2 I , Tg^ = ( v^/2 -1/2 

/ \ / ' ■ ' \ /1/2 v^/2 

= 1 \, =\ -v^/2 -1/2 , T^=\ 73/2 -1/2 





\ 1^ 



(2.17) 



which are block diagonal. It is clear from the definition that the basis is obtained 
from the complex basis by the P5 or P4 phase rotation. Note that, unlike in the complex 
basis, all the representation matrices are real- valued. This is why we call it the real basis. 
Due to this reality of matrix elements of T^^, two types of tensor products are possible for 
5*3 doublets. In the real basis of ()2.1fi|l . the one tensor product of two doublets is 

= (^V^0, V^W)2 +(V^W)u + (V^V)is, (2.18) 

= (^l02 + V'l^i, i^[<Pi - 4^)1 + i4h - 4<Pi)iA + i4<Pi + 4<p2)is- (2.19) 
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Another consistent product can be defined by using transposition of instead of ip'^ . 
This choice is possible since all the group elements are expressed in terms of real numbers. 
In the language of the complex basis, the product defined with daggers corresponds to 
(|2.4p and that with transpositions to (j2.7|) . The difference between these two types of 
products becomes important in the case that S'3-doublet fields transform non-trivially 
under other symmetries than 5*3 . In particular, this is indeed the case for the standard- 
model fermions and Higgs bosons. The real basis has often been used in the literature ^3] . 
For completeness, we comment on another real basis which is defined from T-"^, instead of 
T^^ in (imni . The matrices are now given by T^^ = U'^TfU and their explicit forms 
for 3a are ()2.17|) by changing the signs of Xfse (i.e., T^^^ — > — T^gg). Therefore the 
non-trivial tensor product becomes ip x (j) = (ip'^a^cj), — '?/'Vi0)2 + (^^^cr20)iA + (V'V)is 
that with ijj^. 

Finally we show the singlet components contained in the products of more than two 
doublets. In the real basis, they are given by (up to the fourth order) 

X D iJi(f)i + ^/'202, (2.20) 

i;X(f)XLp D V^l0lV52 + V'l02V5l + V'201</'l - V'202V52, (2.21) 

i>x(j)X(pxx 3 (V^i0i + V'202)(v5ixi + V52X2), (V'lV^i + ^2V52)(0iXi + 02X2), 

(^1X1 + V^2X2)(0lV5l + 02V52), (^"102 - ^20l)(V5lX2 " 'f2Xl), 
{ipl(p2 - V^2V5l)(0lX2 - 02Xl), i^lX2 - V'2Xl)(0lV52 " 02V5l)- 

(2.22) 

In the right-handed sides, the components ipi, (pj, " " " can be replaced with 0|, 0^, ■ ■ • . 
Namely, (V'i0i + "0202), ("0101 V^2 + 0'i02V'i + "0201 V^i - 0'202V^2), and others are also S3 
singlets. This is due to the fact that the representation matrices Tj^ are real in the real 
basis. 



3 Invariant matrices 

Various forms of mass textures of quarks and leptons have been known to be phenomeno- 
logically viable As discussed in the previous section, 5*3 has three types of irre- 

ducible representations; 2, 1a and Is. We would like here to examine what combinations 
of 5*3 representations for matter fields lead to those mass textures. Higgs fields may also 
transform non-trivially under the S3 symmetry and play significant roles for realizing tex- 
tures forms. Since there are three repetitions of matter fields, various assignments of S3 
representations are available, where the first two generations constitute a doublet and so 
on. These include a possibility to realize asymmetrical forms of mass textures and give rise 
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to physically observable effects such as flavor- violating processes in future experiments. In 
particular, as mentioned in the introduction, vanishing matrix elements lead to interesting 
phenomenological consequences. Such zero elements could be obtained in the framework 
of non-abelian discrete flavor symmetry by suitably taking Higgs representations and their 
VEV forms, which are calculable by analyzing scalar potential. Natural forms of VEVs 
generally depend on the 5*3 group basis in which mass matrices are described. At this 
stage, explicit forms of mass textures are governed only by discrete flavor symmetry. In 
some cases, however, additional implementation might be needed to have more control, 
in particular, for fermion mass hierarchy being naturally realized. In the present frame- 
work, the key ingredients for model construction are to select (i) 5*3 group structure, (ii) 
representations of matter flelds, (iii) Higgs profiles (representations and VEVs), and (iv) 
extra symmetries. Let us first comment on these issues in some details. 

(i) ^3 group structures: As mentioned in Section 1231 physical consequences of a mass 
texture potentially depend on the group basis adopted in constructing S3 models. It is 
noted that a choice of specific basis does not affect physical results as long as the fiavor 
symmetry is unbroken: apparent basis dependence, e.g. different forms of mass matrices, 
is only due to a choice of fiavor basis. The S3 invariance guarantees the same spectrum 
without regard to basis choices. However the predictions for generation mixing might 
be different. This is because, in the standard model, there already exists a basis which 
defines the generation structure, namely the interaction basis where the weak current 
interaction is fiavor diagonal. Once one picks up a group basis from some model-building 
perspective, the relabeling of fiavor indices by S3 transformation gives physical meaning 
on generation structure. Another important basis dependence appears in breaking the 
fiavor symmetry. Since any kind of fiavor symmetry has not been observed in the low- 
energy regime, in principle any form of symmetry-breaking patterns is possible. In realistic 
model construction, some guiding principles are often adopted, such as simplicity and/or 
dynamical justification. In either case, symmetry-breaking parameters depend on the S3 
basis. While the breaking parameters take a simple form in one basis, they are rotated to 
a complicated form in other general bases, which form seems to be completely unnatural. 
In this way, the choice of fiavor-group basis may have physical consequences if three- 
generation fermions are assigned to (pseudo) triplets, and therefore is an important factor 
in constructing models with fiavor symmetry. 

Another issue is what types of 5*3 symmetries are involved into the theory. For example, 
with only one S3 symmetry, both left- and right-handed fermions (and also Higgs bosons) 
transform under the same S3. On the other hand, one may easily imagine that three- 
generation fermions have non-trivial charges of separate 5*3 groups. A well-known example 
of the latter case is the democratic mass texture [H] realized by S3L x 5*3/? symmetries. 
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(ii) Representations of matter fields: In this paper, 5*3 is introduced to control the 
flavor structure of quarks and leptons. Three-generation matter fields generally belong 
to non-trivial representations of such flavor symmetries. A charge assignment often as- 
sumed in the literature is that the first and second generations make up a doublet. This 
assignment is adopted to account for several phenomenological issues. First, the flavor 
symmetry invariance suppresses flavor- violating effects between the first and second gen- 
erations, which effects have been tightly constrained by various experimental results in 
supersymmetric theory ^7]. Second, if the third generation has different flavor charges 
from the other twos, they are appropriate to have larger masses. It is however noted 
that all other flavor charge assignments are equally allowed. For example, a phenomeno- 
logically interesting alternative is that the second and third generation leptons make an 
5*3 doublet. This is motivated by the recent observations of neutrino flavor mixing be- 
tween the second and third generations. Moreover it could account for the lightness of 
the first-generation fermions. In what follows, we show that representations of matter 
fields, including left-right asymmetric assignments, are useful for obtaining various types 
of mass matrices. 

(iii) Higgs profiles: Phenomenologically indirect but important possibilities arise for 
the profile of Higgs fields. If S'[/(2)iy-doublet Higgses are in non-trivial representations of 
5*3, Yukawa couplings can be described by renormalizable operators. Higher-dimensional 
operators including multiple Higgs fields are suppressed by a large cutoff scale and give 
negligible corrections to Yukawa couplings, which is a nice feature in a sense that ev- 
erything is described within the renormalizable level. This approach however requires 
that hierarchically small values of Yukawa couplings for the first two generations must 
be fixed by hand as in the standard model. Moreover, due to the existence of multiple 
5'f/(2)iy-doublet Higgses, naive gauge coupling unification is spoiled, and Higgs-mediated 
flavor-violating effects might not be negligible even at tree level. An alternative choice of 
Higgs charges is that S'f/(2)vi/-doublet Higgses belong to the flavor singlet. In this case, 
Yukawa interactions are effectively derived from higher- dimensional operators which can 
be made invariant by introducing appropriate scalars <l>'s in non-trivial 5*3 representations. 
The VEVs of $ break the flavor symmetry and generate trilinear Yukawa couplings below 
the breaking scale. Since the ratio ($) to a cutoff scale gives a unit of Yukawa hierar- 
chy of quarks and leptons, 5*3 should be broken at a high-energy scale below the cutoff. 
This situation resolves the above-mentioned problems of S'3-charged SU{2)w Higgs dou- 
blets: Yukawa hierarchy is explained by controlled higher-dimensional operators, gauge 
coupling unification is preserved, and Higgs-mediated flavor violation is suppressed by a 
large S'3-breaking scale. As an imprint of such high-scale flavor symmetry, new sources 
of flavor violation could arise from renormalization-group evolution below the symmetry- 
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breaking scale. For example, if the theory is super symmetrized, flavor- changing couplings 
are generally induced, and their magnitudes depend on the dynamics of supersymmetry 
breaking. While the flavor violation tends to be small as Kahler terms are limited by 
flavor symmetry, it may be observable, e.g. in the gravity mediation scenario ^1]. Note 
that this type of flavor violation is negligible in the case of S'3-charged Higgs bosons as 
long as the flavor symmetry remains intact at low-energy regime. 

The 5*3 property of Higgs flelds is also relevant to the group basis. That is the form of 
VEVs and their naturalness in the sense of 't Hooft. For example, if a pseudo-singlet fleld 
develops a non- vanishing VEV, S3 is broken down to a subgroup Z3. In the limit of other 
VEVs being zeros, low-energy effective theory still has the residual invariance. In fact, 
for realistic cases, some S3 doublets have nonzero VEVs in order to give non-trivial flavor 
structure. Then the maximal residual subgroup is 6*2. Notice here that the S'2-invariant 
forms of VEVs depend on the group basis. It is found from (j2.3|) and (|2.17|) that such 
technically natural VEV of S3 doublet is proportional to (1, 1)* in the complex basis and 
(0, 1)* [or (1, 0)*] in the real basis. The former is available to realize large flavor mixing 
and the latter is useful for generating textures with vanishing elements. Since the flavor 
symmetry is completely broken at low energy, the group basis might be chosen so that 
symmetry-breaking forms seem as natural as possible. Finally, there is another form of 
doublet VEV which is often utilized in the literature. That is (x, 1)* with x ^ 1, which 
arises from a linear combination of the above two types of VEVs, but it is nothing but 
parameter tuning. 

(iv) Extra symmetries: The S3 flavor symmetry does not fully explain the mass 
hierarchies of quarks and leptons. A hierarchical order of ^s-breaking VEVs requires 
different order of couplings whose origin is generally unclear. An attractive way to dy- 
namically justify hierarchical couplings is to introduce extra symmetries into the theory. 
As for Yukawa couplings, a well-known example is the Froggatt-Nielsen mechanism 
where extra symmetry controls the orders of Yukawa couplings. They are generated in 
low-energy effective theory via decoupling heavy flelds, i.e. higher- dimensional operators 
whose coefficients are naturally given by the fundamental scale of the theory. 

In the following, we present several examples of mass matrices which stem from the 
5*3 flavor symmetry. In almost the cases, three-generation fermions belong to non-trivial 
triplet representations: two of them make up an 5*3 doublet and the other is a (pseudo) 
singlet. On the other hand, appropriate representations of Higgs flelds are chosen to have 
non- vanishing elements in 5'3-invariant Yukawa matrices. Mass textures will be expressed 
in the complex basis unless particularly mentioned. The expressions in the other bases 
are easily obtained by the unitary basis rotations deflned in the previous section. 
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3.1 A single ^3 

First we study the case that both left-handed fermions ipn and right-handed ones tpRj 
= 1,2,3) transform under a single S3 group. This case is also straightforwardly 
applied to Majorana mass terms with the identification ipi = ipR. In the following, we 
take a charge assignment that {4'li,4'l2) and {i'Ri,4'R2) are 5*3 doublets and the other 
fermions are in (pseudo) singlet representation. This does not lose any generalities since 
mass matrices for other charge assignments are obtained by flavor rotations. As for Higgs 
fields, all the three types of irreducible representations are available for giving nonzero 
matrix elements; a doublet Hd = {Hi,H2), a pseudo singlet Ha, and a singlet Hs- Here 
Hd,s,a are interpreted as either S'f/(2)vi/-doublet elementary scalars or products of some 
numbers of fields [see, e.g. the comment (iii) Higgs profiles in the above]. 

3.1.1 The general case 

Dirac mass terms flip the chirality of fermions. Given that iIjl^, and ipR^ are singlets, the 
most general S'3-invariant Dirac mass matrix is described as 

-Coirac = -IpRiMij^pLj + h..C., (3.1) 

faHs + a'H*s + bHA + b'H\ CH2 + c'H* dHi + e'Hl \ 

M = cHi + c'H* aHs + a'H* - BHa - V H\ dH2 + e'Hl , (3.2) 

V eH2 + d'H* eH, + d'H; fHs + f'H*) 

where a, a',b, ■ ■ ■ , /, /' are independent coupling constants. The generic form of mass 
matrix ()3.2|) is simplified if Hj:),s,a belong to complex representations of other groups 
than 5*3: either Hx or corresponding H* is dropped out in each element of the generic 
matrix. This is indeed the case for the SU{2)w Higgs doublets in the standard model. 
Another charge assignment is that the pseudo singlet representation 1a is adopted for the 
third-generation fermions. For example, when ipi^ is a pseudo singlet, the third column 
of (j3.2j) is modified so that H2 and Hs are replaced with —H2 and Ha, respectively. 

The general form of Majorana mass matrix is described by identifying ipi = '4'R = 
It is noted that the tensor product ()2.7|1 should be used in constructing Majorana mass 
term in the complex basis. Thus the S3 invariance leads to 

-^^Majorana = V'i Mij Ipj + h.C, (3.3) 

/ aHi + bH* cHs + c'H*s dH2 + eHl \ 
M = \ cHs + c'H* aH2 + bHl dH^ + eH* . (3.4) 
\dH2 + eH* dH^ + cH* fHs + f'H*J 

Similar to the above case of Dirac masses, either H^j. or H* is removed in each matrix 
element when H^ has some (complex) quantum charge other than that of 5*3. Due to 
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the difference of 5*3 tensor products used for Dirac and Majorana mass terms, tlie result- 
ing flavor structures in ()3.2|) and ()3.4|) are rather different. This fact could provide an 
interesting possibility for realistic model construction of flavor. 

3.1.2 Supersymmetric case 

Supersymmetry invariant fermion masses come from the superpotential, which is a holo- 
morphic function of chiral superfields. That is, superpotential terms are written in terms 
of chiral superfields, which contain fermionic components with a certain chirality, e.g. 
left-handed fermions. In this case, right-handed fermions are introduced as charge con- 
jugations of left-handed ones. As in the general case, suppose that the chiral superfields 
of first two generations, (^'li,^L2) and {"^ m,"^ R2), are S3 doublets, and \E'_r3, \E'l3 are 
singlets. It should be noted that this assignment leads to the right-handed fermions 
{4'Riy4'R2) being an anti-doublet. Therefore the product (j2.7p is applied to constructing 
5'3-invariant superpotential. The most general supersymmetric mass term is thus given 

by 

VrDirac = "^mMij^Lj, (3.5) 

/ aHi bHs + cHA dH2\ 
M = IbHs- cHa aH2 dHi . (3.6) 
V eH2 eH, fHsJ 

In the case that \E'l3 is assigned to a pseudo singlet, one needs the replacement H2 — ^ —H2 
and Hs — »■ Ha in the third column of ()3.6|) . Similarly, the S'3-invariant Majorana mass 
term is easily found by symmetrizing the matrix ()3.6j) . namely, by setting the couplings 
as c = and d = e. This does not significantly modify the flavor structure, unlike in the 
non-supersymmetric case. 

3.1.3 Relation to the democratic mass matrix 

We have descried the mass matrices in the complex basis of the 5*3 group. Let us here 
comment on the relation to the so-called democratic form of mass matrix in which all the 
matrix elements have equal magnitude. It has a simple 5*3 derivation in the democratic 
basis defined in Section !^. 3. 11 As we explained before, these two group bases are connected 
by the unitary rotation U in (I2.13p . A simple way to recover the democratic mass matrix 
is to introduce only an S'3-singlet Higgs field Hs- As seen from ()3.2|) . that corresponds to 
a flavor-diagonal matrix in the complex basis. By rotating the matrix to the democratic 
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basis, it turns out to be 




M = \1 1 l\hs+\ 1 \h's, (3.7) 



where hs = [(/ - a)Hs + (/' - a')Hg]/3 and h'g = aHs + a'H^. It is not hard to see the 
5*3 invariance of the above two matrices in the democratic basis. The democratic mass 
matrix is thus found to be derived from an assumption that only one matrix element is 
dominant in the complex basis. Therefore a single 5*3 symmetry cannot ensure the flavor 
democracy: even with a single Hs field, the democratic ansatz is generally disturbed. As 
seen from the representation matrices ()2.14p . the democratic basis deals with the three 
indices of a (pseudo) triplet equivalently, and is not suitable to discuss the decomposition 
to irreducible representations. 



3.2 SsL X SsR 

For controlling chirality-flipping operators, one can utilize a flavor symmetry under which 
left-handed and right-handed fermions transform separately, that is, the S^l x S^ji symme- 
try. Without loss of any generalities, we assume that the first two generations {ipLi,ipL2) 
and {i'Ri,i'R2) belong to the doublet representations of 5*3^ and S^r, respectively. Since 
there exist three irreducible representations for the 5*3 group, fermion bilinear terms trans- 
form in nine ways under S^lX S^r. The corresponding nine types of scalars which ensures 
the flavor invariance are denoted by Hij = D,S,A), where an obvious notation has 
been used, e.g. Hda means a doublet under S^l and a pseudo singlet of S^r. As mentioned 
before, the symbols Hij stand for either S'f7(2)vi/-doublet elementary scalars or products 
of some fields with appropriate charges. 



3.2.1 The general case 

For an illustration, we assume that the third-generation fermions ipL3 and ipR3 are singlets 
of ^3^ and SsR, respectively. In case that they are pseudo singlets, the subscripts S of 
Higgs fields should be replaced with A in the following expressions. The most general 
Dirac mass matrix is made symmetry invariant by introducing Hdd, Hds, Hsd, and 
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Hss, and is given by 



-^Dirac = -IpRiMijlpLj +'^-C., 



(3.8) 



M 



12 '^(-f^Z)Z))l2 + b{H£)£))ii 

+ c{H^^)21 + d{H^^)22 +ciH^^)22+d{H^^)21 

a{HoD)2l + b{HoD)22 0,{Hdd)22 + i>{HoD)2l 

+ c{H^^)ii + d{H*o)i2 +c{H*^^),2 + d{H*^^)n 

V g{Hns)2 + h{H*s)i g{Hns)i + h{H*s)2 



e{HsD)2 + f{H*^), 



jHss+fH*s I 



\ 



(3.9) 



where a,b, ■ ■ ■ , j, j' are the couphng constants. The Higgs fields either with or without 
asterisks are dropped out in each matrix element if they have some quantum numbers of 
other symmetries than 5*32, x S^^. The possible Majorana mass term of tpL (or ^i?) is 
written in the same way as ()3.4j) by including other if's with appropriate charges. 

The Dirac mass term in supersymmetric theory is given by the superpotential which is 
a analytic function of superfields with definite chirality. If we take the same fiavor charge 
assignment as above for the three-generation superfields \E'Lj and "^rj, the most general 
form of mass matrix is 



3.2.2 Examples 

It is found from (|3.9p that the general form of Dirac mass term is rather complicated. 
Combining with simple assumptions, we here present several examples where phenomeno- 
logically interesting forms of mass matrices are obtained, in particular, by choosing rele- 
vant Higgs contents. 

The first example is the introduction of a single elementary scalar H in the (pseudo) 
singlet representation of both S-^l and S^r symmetries. According to the 5*3 charges of 
'ipL3,R3, a relevant representation of the scalar becomes that of Hij with i,j = S or A. In 
any case, only the 3-3 element of mass matrix is allowed; 



(3.10) 




a{HDD)22 0'{Hdd)21 b{HsD)2 

a{HDD)i2 a{HDD)n b{HsD)i 
c{Hds)2 c{Hds)i dHss 




(3.11) 




(3.12) 
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If turning to the democratic basis, we have 



M 



3 



1 



1 
1 
1 



1 
1 
1 



1 



) 



aH. 



(3.13) 



Both of these matrices have only one nonzero eigenvalue and the same mass spectrum. 
However the resulting flavor mixing structures are clearly different: the matrix ()3.13p 
induces large flavor mixing, while ()3.12|1 has no mixing between the third and the flrst- 
two generations. As we noted before, this may be a physical difference because, in realistic 
models, there exists more than one sector whose relative basis differences do affect flavor 
structure such as the quark mixing angles. 

The second example is not to consider that elementary scalars Hij give all trilin- 
ear Yukawa couplings, but to work with some products of scalars in lower-dimensional 
representations. As an example, let us introduce three scalar flelds Hd = {Hi,H2), 
H'jy = {H[, H^) and Hs whose representations of {S^l, S^r) are (2, Is), (Is, 2), and (Is, Is), 
respectively. With this fleld content and the matter representation as above, all entries 
in the Dirac mass matrix can be fllled up with non-renormalizable operators. In super- 
symmetric theory, the mass matrix is now given by 



On top of economical fleld content, this example has several useful properties for con- 
structing flavor models of quarks and leptons. First, the matrix ()3.14j) has a vanishing 
determinant and therefore provides a compelling dynamical reason for the observed tiny 
masses of flrst-generation fermions. Secondly, the effective Yukawa couplings for the 
first-two generations become naturally small. This is because the 5*3 invariance requires 
that they come from higher-dimensional operators suppressed by some large mass scale. 
Thirdly, the hierarchical fiavor structure is easily attained with a smaller number of scalar 
VEVs. It might often lead to some relations among mass eigenvalues and mixing angles of 
quarks and leptons. One can make use of these interesting features with a smaller number 
of representations of scalar fields than ()3.9|) . 



We have so far discussed the case that both left- and right-handed fermions have non- 
trivial charges of discrete fiavor symmetry. In this subsection, we comment on a possibility 
that either left- or right-handed generations is insensitive to fiavor transformation. 




(XH2H2 (XH1H2 
(1H2H1 clHiH'^ 
CH2 cHi 



hH'2 
bH[ 
dHs 




(3.14) 



3.3 Singlet flavor 
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Let us consider a single 6*3 symmetry under which left-handed generations transform 
non-trivially but right-handed ones do not. In a similar way to the discussion in Sec- 
tion EHHl the generic types of representations, Ho = {Hi,H2), Hs, and Ha, are taken 
into account. If one takes a charge assignment that {'4'li,'4'l2) is a doublet, the most 
general Dirac mass matrix is given by 



-^Dirac = -lljRiMijlpLj +h-.C., (3.15) 

aH2 + bHl aHi + bH* gHs + g'H* \ 
M = \ cH2 + dH* cHi + dH; hHs + h'H*s\. (3.16) 
eH2 + fHl eH^ + fH* jHs + fmJ 

In each matrix element, a symbol H either with or without asterisk should be dropped if 
H is accompanied with some charges other than that of S3. In supersymmetric theory, 
the mass matrix is simplified to 

/ aH2 aHi gHs\ 
M = \cH2 cHi hHs . (3.17) 
\eH2 eH, jHsJ 

If only a singlet Higgs field exists, that results in 

^9 9 9' 

firrfjl ^ \ h h h\ Hs (3.18) 

in the democratic basis. This form of Dirac mass matrix has been discussed in the so-called 
lopsided models. Similar results are also obtained with the 1a representation instead of 
Is- 



4 Mass textures from ^'3 

On the prescription described in the previous sections, we will perform several construc- 
tions of mass textures by use of flavor S3 symmetry. That includes well-established forms 
of mass matrices which have been discussed in the literature. In realistic flavor models 
for quarks and leptons, some combinations of texture forms are usually assumed, and it 
is therefore meaningful to examine whether they can be simultaneously reproduced by 
horizontal symmetry. 

It is found in Section IHl that the key ingredients for 5*3 model building are to define 
matter and Higgs profiles and, if needed, to introduce additional symmetry. Among such 
further symmetries, we will focus in this work on U{1) flavor symmetry to dynamically 
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realize fermion mass hierarchy 120] • It is, however, highly non-trivial to assign rel- 
evant f/(l) quantum numbers. This is because an 5*3 doublet, e.g. the first and second 
generations in the above matrix examples, must have the same U{1) charge and it does 
not provide mass hierarchies between them. Moreover, a naive charge assignment often 
leads to a prediction that classical mass hierarchy realized by fiavor symmetry is disturbed 
by higher-dimensional operators. These facts are typical features of 5*3 fiavor models ac- 
companied by additional U{1). In models without discrete fiavor symmetry, U{1) charge 
assignment generally has wider fiexibility [201 • On the other hand, models with only 5*3 
symmetry have to do with fermion mass hierarchy by arbitrarily tuning parameters such 
as Yukawa couplings and Higgs VEVs. In the following, we illustrate several examples 
of mass textures derived from 5*3 and assistant U{1) fiavor dynamics, and also mention 
how to cure the above problems of charge assignment. It is noticed that mass hierarchy is 
dynamically realized in a similar way with a discrete subgroup of the fiavor U{1) symme- 
try. For example, all the hierarchical mass textures we discuss below can also be obtained 
by Zn subgroup with appropriate (enough large) N and the same quantum numbers as 
in the U{1) case. A smaller choice of would be possible and interesting from a view- 
point of brevity. In this case, the problem of fermion masses can be handled with fiavor 
symmetries that are entirely discrete. 

In what follows, we consider the cases that the fiavor symmetries are broken at some 
high-energy scale, and renormalization-group running down to low energy should be taken 
into account if one obtains precise values of coupling constants, once the model below the 
symmetry-breaking scale is specified. However, it is not hard to see that, in all examples 
we discuss below, the running effects do not change qualitative results and can safely 
be dropped. The analysis in this section is performed in the models with a single fiavor 
5*3 symmetry and the results are described in the complex basis, unless we particularly 
mention it. 

4.1 Nearest neighbor form 

The first example of models includes the mass texture proposed by Fritzsch pp. We 
assign non-trivial charges to three-generation left- and right-handed fermions ip^i and 
ipRj {i = 1,2,3). They are given by (^li,V'L2) + V^ls and {i'Ri,i'R2) + i^Rs, namely, the 
first and second generations make up 5*3 doublets and the third ones are singlets. 

For the Higgs profile, we introduce an S'3-doublet Hd = {Hi, H2) as only a scalar with 
non-trivial 5*3 charge. As mentioned in Section 121 Hd has two possibilities concerned 
with the electroweak charge. We here take Hd as a singlet of the electroweak gauge 
symmetry, and accordingly utilize the usual 5'f/(2)vy-doublet Higgs h. Advantages of this 
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choice are the suppression of flavor- changing rare processes, the preservation of gauge 
couphng unification in supersymmetric theory, and so on. A key point is that the VEV 
of the 5*3 doublet scalar is assumed to take the form (//d) = {{Hi), 0). This form can be 
dynamically justified by analyzing the scalar potential, as we will show in later section. 
Another type of VEV, {Hd) = (0, {H2)), just gives the case easily found by exchanging 
the up and down components in 83 doublets. 

The next step is to impose extra symmetries. As a simple example, we here assume a 
Z2 parity acting non-trivially on 6*3 doublets ("doublet parity"). We also have flavor U{1) 
symmetry in order to control mass hierarchy and mixing among the three generations. 
Notice that due to the assignment of these non-vanishing charges the resultant mass 
matrices are simplified. That is, either a Higgs field or its complex conjugate can appear 
in each matrix element. At this stage, the 3x3 mass matrix for ipL,R is found to be 

\ 

b{Hi) h, (4.1) 
c{Hi) d J 

where a, b, c, and d are the 0{1) coefficients which include coupling constants and, if 
any, scalar fields with non-vanishing VEVs. Throughout this paper, 0{1) coefficients are 
denoted where the fundamental mass scale of theory is taken to be unity. The texture form 
()4.1|) was first adopted by Fritzsch pP for quark mass matrices. It is noticed that the parity 
leads to some of vanishing matrix elements at tree level. An alternative parity assignment 
is viable where ipis, ipm and Hr, have negative parity. A more interesting possibility is to 
incorporate supersymmetry in the model. In this case, vanishing matrix elements could 
be due to the holomorphicity of superpotential, combined with U{1) quantum charges for 
realizing mass hierarchy, i.e. supersymmetric texture zeros. 

To generate fermion mass hierarchy, we use the fiavor U{1) symmetry whose quantum 
charges are assigned as in the table below: 







h Hd ^ 


U{1) 


X x' y y' 


Q z -I 


Z2 


- + - + 


+ - + 



We have introduced a scalar field tp to describe symmetry-invariant higher-dimensional 
operators which effectively induce mass terms below the U{1) breaking scale {(p). The 
U{1) charges are normalized by letting Lp take a unit charge, and h is assumed to be 
neutral in order to fix overall mass scale. As a result, the orders of magnitude of the 
matrix elements in ()4.1|) becomes 

\ yx+y'+z+a )x'+y' I 
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where we have defined {ip) = A and (Hi) = A" (a > for a consistent theory below the 
fundamental scale). 

The eigenvalues of (Q are easily found; mi ~ hx^+y-2{z+a) ^ _ ^^^+y+2{z+a) ^ 
and ~ hX^'^^' . To obtain a hierarchy mi <^ 1712, the condition z + a < is naively 
needed. It should be, however, noticed that this condition causes a problem that higher- 
dimensional operators involving Hd give larger effects and are not necessarily negligible. 
We will in the below enumerate possible resolutions to the problem; (i) to set 2; + a > 0, 

(ii) to take z < and make higher-dimensional operators negligible by supersymmetry, 

(iii) to tolerate some of higher- dimensional operators, (iv) to impose extra symmetry, (v) 
to change the S3 representations of fermions, and (vi) to introduce ^s-singlet scalars. 

(i) If one chooses z + a > 0, problematic non-renormalizable operators become irrele- 
vant. This may be the simplest solution in the viewpoint of U{1) charge assignment. It 
is found from the matrix form ()4.2|) that the mixing angle between the first and second 
generations becomes 7i/2. Note here that the 5*3 freedom cannot be used to reorder the 
mass eigenvalues because the 5*3 group basis has already been fixed such that Hd takes 
a particular form of VEV. While we know the generation mixing in the quark sector is 
small, the 5*3 realization of the Fritzsch ansatz can be consistent (for z + a > 0), provided 
that the up and down quark sectors have almost the same fiavor structure. Namely, If 
both the up and down sectors employ the matrix form (|4.2p . the label exchanging effects 
are cancelled out between the two sectors, and the quark mixing angles become small 
of powers of A. Such a situation is similar to the case of the democratic quark mass 
matrices |14j . 

(ii) If one chooses z < 0, higher-dimensional operators can be forbidden by holomor- 
phicity in supersymmetric theory. For this being achieved, it is a simple assignment that 
all the U{1) charges except for Hd and (f are positive in order not to give any higher- 
dimensional terms. Since the doublet scalar now develops a VEV in the up component, 
the down component of doublet in the product {HdY becomes nonzero, and the singlet 
component in {HdY does [see the tensor product (j2.7|) ]. Consequently, higher-order terms 
involving the products of Hd might spoil the first-order analysis, because Ho has a nega- 
tive U{1) charge {z < 0). In fact, the 1-3, 3-1, and 2-2 elements in the matrix (j4.ip receive 
non-negligible contributions from [HoY- The contributions to the 1-3 and 3-1 elements 
are forbidden by the doublet parity. We however find that higher-order contribution to 
the 2-2 element is difficult to be suppressed as long as the 2-3 and 3-2 matrix elements 
are allowed. A simple way to remedy this last problem is to add some extra symmetry, 
otherwise to apply the option (iii). 

(iii) One, in some sense, negative choice is to abandon the exact Fritzsch ansatz and 
put up with some contribution from non-renormalizable operators. As mentioned above. 
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a non-vanishing 2-2 element often appears even if the parity invariance is imposed. One 
can therefore choose as the third option that higher-order terms are generally forbidden 
by parity and/or supersymmetry while the 2-2 element is not. For example, in supersym- 
metric approach, the U{1) charges are needed to satisfy the mild constraints x + y + z < 0, 
X + y' + z > 0, x' + y + z > 0, and z < 0. If this is the case, we obtain a mass matrix 
of the form (j4.ip corrected by a nonzero 2-2 element from a higher-dimensional opera- 
tor involving {HjjY. Such a type of mass texture has been recently discussed to be 
suitable for solving fermion mass problems including the neutrino physics. 

(iv) If one chooses to introduce more additional symmetries, harmful higher-dimensional 
operators might be removed. However such an operation generally reduces to complicate 
the models and involve uncontrollable factors, which make the models unfavorable. 

(v) Contrary to the above options (i)-(iv), one can choose to extend the model to 
include more fields in other representations of flavor symmetry. Let us consider an addi- 
tional Higgs Ha of pseudo 5*3 singlet. The representations of matter fields are accordingly 
changed to {i'L2,4'L3) + i^Li and {ipR2^'>Pm) +'^rii i-e. the second and third generations 
are doublets. It is easily found by constructing Ss-invariant terms that one still has a 
mass matrix of the Fritzsch ansatz, provided that the non-vanishing VEVs are given by 
(if/)) = (0, (-^2)) and {Ha)- An important difference between this and the above models 
is whether the parity symmetry is needed or not to suppress undesired matrix elements. In 
the model here, the Fritzsch ansatz is obtained without imposing any parities. To make 
the 1-1 element negligible, it is sufficient to take appropriate U(l) charge assignment, 
since the first-generation fermions now belong to different 5*3 representations from the 
others, and have different U{1) charges. Note that the example here may not be applied 
to left-right symmetric cases due to the anti-symmetric matrix elements generated by a 
pseudo singlet (i.e. M23 = — M32). However asymmetrical forms of mass texture (zeros) 
could provide phenomenologically interesting possibility. 

(vi) Another choice of additional scalar fields is a singlet Higgs Hs- A reason to 
introduce such singlet scalars is to suppress bare Yukawa couplings [e.g. a and d in the 
matrix 1)4.11) ]. They are induced from other operators involving 6*3 singlet scalars with 
non-vanishing f/(l) charges. Once the singlet scalars obtain VEVs, the resultant mass 
matrix takes the same form as ()4.H) . but fiavor- invariant non-renormalizable operators 
are different due to the non-vanishing charges of singlet scalars.^ Let s be the U{1) charge 
of Hsi which must be positive as will be seen below. There are three possible ways to 
include the Hs scalar: (a) The 3-3 element comes from an operator involving Hs- That 
needs the charge conditions x' + y' < Q and x' + y' + s > 0. Moreover, for the mass 

Hn case that S'J7(2)iy-doublet Higgses belong to non-trivial representations of 5*3, such effects of Hs 
(the suppression of bare Yukawa terms) can always be taken into account. 
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hierarchy being reahzed, an additional condition 2{z + a) < s + as is required, where 
(Hs) = A"^ It is interesting that the operators which contribute to the 2-2 element is 
automatically suppressed, and one does not need to rely on any additional symmetries. 
Furthermore if one chooses s + ag < §(-2 + a) [or s + a^ < 3{z + a)], corrections to the 1-3 
and 3-1 [or 1-1] elements become negligibly small. It can be checked that all the above 
charge conditions are easily satisfied, (b) In this case, the charge conditions x + y < 
and x + y + s > are needed for Hs to appear in the 1-2 and 2-1 elements. A proper mass 
hierarchy is realized when 2{z -|- a) < s -|- a^, and the 1-3 and 3-1 elements are negligibly 
small a s + as < ^{z + a). Unlike in the ), these charge conditions do not suppress 

higher- dimensional contributions to the 1-1 and 2-2 elements, so some symmetry should 
be imposed, (c) All the 1-2, 2-1, and 3-3 elements contain the Hs field. That requires 
X + y < 0, x' + y' < 0, X + y + s > 0, and x' + y' + s > 0. Moreover one should take 
z + a < s + as in order for fermion mass hierarchy to be preserved. With these charge 
conditions, all the operators concerned with the 2-2 element are automatically suppressed. 
It is, however, found that the 1-1 element generally receives sizable contribution unless 
extra symmetry is imposed. 

In this way, the nearest-neighbor form of mass matrices, including the well-known 
Fritzsch ansatz, is realized in the framework of 5*3 fiavor symmetry. Moreover the mass 
hierarchy among the three generations is also achieved by incorporating an additional 
C/(l) symmetry. An important and non-trivial issue is whether U{1) charges can be 
assigned so that they are compatible with the non-abelian fiavor symmetry. We have 
shown typical examples of assignments which not only produce mass hierarchy but also 
suppress non-renormalizable operators which tend to disturb the mass hierarchy in the 
first-order estimation. 

4.2 Next-nearest neighbor form 

In most models with fiavor symmetry and also in the previous subsection, the first two 
light generations are assumed to compose S3 doublets. While such an assignment may be 
favorable to some phenomenological issues e.g. for suppressing fiavor- changing processes, 
it is not necessarily the unique choice for 5*3 charges. One may easily imagine different, 
but somewhat unfamiliar, 5*3 fiavor structures. In the following, we show that 5*3 models 
with such twisted generations are also relevant to constructing realistic fiavor theory of 
quarks and leptons. As a simple example, we here discuss the ansatz for mass texture 
proposed in [3], which type of texture suggests that the first and third generation fermions 
make up 5*3 doublets. 

The procedure is completely parallel to that in the previous subsection. The key 
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ingredients are the representations of matter and Higgs fields, the profile of Higgs VEVs, 
and additional symmetries. Consider the matter representation {ipLi,4'L3) + i^L2 and 
(V'-RIjV'rs) + V'-R25 i-e. the first and third generations are 5*3 doublets. We also introduce 
an ^s-doublet Higgs scalar Hj:, = {Hi,H2), whose down component H2 is assumed to 
develop a non-vanishing VEV. The electroweak gauge invariance is implemented by an 
5'f/(2)vi/-doublet scalar h, which is a singlet of S3. The setup leads to the following form 
of mass matrix; 

a \ 

b h, (4.3) 

a c{H2)J 

where 1-2 and 2-1 elements have been suppressed by imposing a Z2 parity concerning the 
second generation; ipL2, ipR2 —i^L2, —i^R2- Similar suppression can also be obtained by 
use of holomorphicity in supersymmetric theory. At this stage, the coefficients a, b, and 
c are supposed to contain coupling constants and scalar VEVs, whose natural sizes are 
0{1). The texture ()4.3|) has the form of the Giudice ansatz |3] for the up quark mass 
matrix. The coefficients a and b should be smaller than c{H2) to properly describe mass 
hierarchy of fermions. That is realized in the present work by introducing flavor U{1) 
symmetry and a charge-compensating scalar ip. Defining U{1) quantum charges as in the 
following table, 
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Z2 
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we have the orders of magnitude of the matrix elements 

/ A^+^ \ 

A^'+^' h, (4.4) 

where {ip) = A, and a parametrizes the VEV of 6*3 doublet {H2) = A°. It is obvious that 
a charge condition 2; + a < is necessary for mass hierarchy without inducing maximal 
generation mixing. Notice that this is similar to the case of the Fritzsch ansatz where 
higher-dimensional operators involving the S3 doublet scalar lead to significant modifi- 
cation of matrix form. Therefore also in the present case, one can apply the resolutions 
discussed in the previous subsection to have natural hierarchy of mass eigenvalues and 
mixing angles. 

4.3 Asymmetric textures 

A more unfamiliar but interesting case is that fiavor charges are asymmetrically assigned 
to left- and right-handed fermions, which generally lead to asymmetrical forms of mass 
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textures. In grand unification schemes, the mass matrix of up-type quarks is often assumed 
to be symmetric (exactly speaking, hermitian), which comes from the fact that, even in 
the minimal SU{5) model, one-generation up-type quarks with both chiralities belong 
to a single multiplet of unified gauge symmetry. However this is not generally the case 
for other Dirac mass matrices. In particular, the present experimental data suggest that 
the leptonic flavor mixing is quite un-parallel to the quark mixing. This asymmetrical 
observation can be compatible with quark-lepton unification, if fermion mass textures 
take asymmetrical forms in the generation space jTU]. There are also some classes of non- 
hermitian ansatze for quark mass matrices [22] , which are consistent with the experimental 
data and cannot be transformed to the symmetric solutions previously found in Ref. jl]. 
It is therefore worthwhile to investigate the dynamical realization of asymmetric mass 
textures with discrete flavor symmetry. A systematic study of asymmetric mass matrices 
for the up and down quarks has recently been performed, particularly paying attention 
to the connection to leptonic flavor mixing [221 • 

Among various types of viable asymmetric textures, we here present an ^3 flavor 
model which predicts the mass matrix proposed in [21]. This texture is relevant to the 
neutrino Dirac mass matrix and is interesting in that large lepton mixing is realized 
without any tuning of couplings, if there is a suitable hierarchy among right-handed 
neutrino masses. The texture indicates neither that the first-two generations are in doublet 
representation nor that left- and right-handed fermions have parallel assignments of 5*3 
charges. That leads us to consider highly asymmetric flavor structure: ('?/'li, V'ls) + i^L2 
and {ipR2,4'R3) + i^Ri- We also have SU (2) ^-singlet scalars, Hs and Hd = {Hi,H2), 
which are 5*3 singlet and doublet, respectively. The down component H2 of the doublet 
Hd is assumed to develop a non-vanishing VEV. The electroweak gauge invariance is 
maintained by introducing an SU {2) w- doublet standard model Higgs h, which is a singlet 
of 5*3. Consequently we obtain the following form of mass texture: 



It is interesting to note that no additional symmetry is required to eliminate unwanted 
nonzero matrix elements, contrary to the previous two examples. The coefficients a, ■ ■ ■ , e 
contain coupling constants whose natural sizes are 0{1). Splitting the sizes of the co- 
efficients is easily obtained, for example, by introducing flavor U{1) symmetry and a 
charge-compensating scalar (p. A typical U{1) charge assignment is given by 




(4.5) 



■ipLl,3 V^L2 V^fl2,3 'ipRl h Hp Hs ^ 



U{\) 



—X y z Q X 2x —1 
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with X > y > and z > 0, and we obtain the mass texture of the form 

The parameter A denotes the ratio of the VEV ((/?) to the fundamental scale. Compared 
to the matrix in a non- vanishing 3-1 element is generated. However one can easily 
find that such a tiny entry does not contribute both to the mass eigenvalues and mixing 
angles, though it is obliged to be included to respect the flavor symmetry. 

It was discussed |E4j that the texture (j4.6|) (without the 3-1 element) could be derived 
by making use of more than two continuous flavor symmetries. By contrast, the present 
approach with discrete flavor symmetry seems somewhat simpler. The texture form is 
totally controlled by a single 5*3 symmetry, and then, the mass hierarchy is determined 
by extra assumption (here the U{1) flavor symmetry). 

Finally let us comment on the corrections from non-renormalizable operators. It is 
found that the above charge assignment suppresses the contribution of higher-dimensional 
operators to the 1-3 and 3-2 matrix elements, and the texture zeros are not destabilized. 
On the other hand, the correction to the 2-1 element could be as large as the other compo- 
nents in the second rows, i.e. (9(A^^^). This result itself is not a disaster but the above zero 
texture should be modifled. That is similar to the cases in the previous subsections where 
higher- dimensional operators involving Hu could give signiflcant effects on the flrst-order 
approximation. Therefore one is able to apply similar resolutions to remedy the problem. 
For example, if the 5*3 doublet scalar Ho has a non-trivial charge of some other symmetry, 
any problematic deformation of mass matrix is completely avoidable. 

4.4 Missing a right-handed neutrino 

Referring to the S^, algebra presented in Section |2l the maximal irreducible representation 
is doublet. The triplet representation is reducible, while it may be suitable to describe 
three-generation fermions. Two fermions with the same standard model charges are also 
well handled with 5*3 symmetry. While we have already known the existence of three 
repetitions of the standard model fermions, there is an idea that only two right-handed 
neutrinos are effectively included j2Sl- In this framework, a missing right-handed neutrino 
might be thought to decouple at a superheavy scale and regarded as a flavor singlet. * The 
remaining two right-handed neutrinos are hence treated as a doublet. In this subsec- 

■''Such a singlet should be Is to preserve syrametry in low-energy regime. The existence of the third 
right-handed neutrino would be needed to reconcile the lepton sector to the quark one in high-energy 
unification schemes. 
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tion, we present a model with three left-handed and two right-handed neutrinos in the 
framework of S3 flavor symmetry. 

We would like here to stress that another important issue of flavor symmetry is to 
simultaneously control different sectors of flavor. For example, physical quark mixing 
angles are determined with a conspiracy of the up and down mass matrices, and low- 
energy Majorana neutrino masses are derived from Dirac and right-handed Major ana 
masses with the seesaw mechanism [26,. In the following, we illustrate simultaneous 
treatment of two different types of mass textures, i.e. the Dirac and Majorana masses of 
neutrinos. A more realistic flavor model, including quarks and charged leptons, will be 
discussed in the next section. 

Among various models with two right-handed neutrinos, we here focus on the texture 
ansatz proposed in j2ZI and its realization with 5*3 flavor symmetry in the neutrino sector. 
The model contains three left-handed leptons V^li,2,3 and two right-handed ones ipRi^2- 
We assign the flavor charges to these fermions such that {'iIjli,iI'L2) and {4'ri,iPr2) are 5*3 
doublets and iIjl3 a singlet (either of the 5*3 singlet representations is possible). As for 
the Higgs sector, at least two types of scalars are introduced for controlling two different 
sectors of mass textures, though not necessarily required. We take, as an example, two 
scalar fields which are 5*3 doublets and the standard gauge singlets; Hd = {Hi,H2) 
and H']j = {H^^Hi^), whose VEVs generate Dirac and Majorana masses, respectively. 
For compensating electroweak gauge invariance, a usual S'f/(2);4/-doublet Higgs h is also 
included. A more economical choice might be to have S'f/(2)vi/-doublet and S'3-doublet 
scalar fields. In this case, however, there may exist some problems, as discussed before, 
that gauge coupling unification must be non-trivially realized and large rates of flavor- 
changing processes generally spoil the models. 

The 5*3 invariant terms induce the neutrino Dirac mass My and the right-handed 
Majorana mass Mr, which read from the generic expressions ()3.2j) and ()3.4j) . 



in [27], if the generation indices are properly exchanged while physical consequences are 
unchanged (except for a sign reversion of as yet unobserved leptonic CP-violating quan- 
tity). The bi-large generation mixing in the neutrino sector is naturally established, 
provided that (Mi,)i2 — (Mi,)i3 and {My)2i — (M,^)23, and that there is a little hierarchy 
between these two combinations or between the mass eigenvalues of Mr [TF] . 




(4.7) 




(4.8) 



where a, ■ ■ ■ , / are the coupling constants. Thus we have obtained the textures discussed 
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It is rather straightforward to incorporate supersymmetry into the above picture. 
Every field is promoted to a superfield which is in the same representations of the standard 
model gauge and fiavor symmetries. The Dirac and Majorana mass terms are described 
by superpotential. Notice that the right-handed neutrinos tp^n are embedded into the 
corresponding superfields in the form ip^i . As we explained in Section ^ this embedding 
modifies the tensor product and slightly changes the texture form. We thus find that the 
5'3-invariant superpotential induces the mass matrices 



Again the neutrino mass textures presented in |2ZI are dynamically realized in a super- 
symmetric 5*3 model. It is easy to see that the coupling constants and scalar VEVs can 
be appropriately chosen for the neutrino physics without any fine tuning. 

5 Grand unification with fiavor Ss 

Based on the above prescription for mass texture, in this section, we present a grand 
unified model of quarks and leptons with S3 flavor symmetry. It is the most important 
point of flavor symmetry that any matrix form from flavor symmetry is valid only in 
the case that there are more than two sectors governed by the symmetry. A well-known 
example is the up and down quark mass textures whose forms are simultaneously altered 
by flavor rotation of S'f/(2)vy-doublet quarks. In other words, any realization of zero 
textures for a single sector cannot be physically distinguished from other freely-rotated, 
generally complicated, matrix forms. In the previous subsection, we discussed a model 
in which two types of neutrino mass matrices are controlled in the same fashion. To 
include the charged lepton sector is straightforward, referring to the general discussion 
of S'3-invariant matrix forms. Towards realistic flavor models of quarks and leptons, we 
here construct an illustrative grand unifled model with S3 flavor symmetry. The model 
has asymmetrical forms of mass textures due to non-trivial assignment of flavor charges, 
which particularly leads to bi-large lepton mixing. While the model presented here is not 
so complete, it suggests the validity of discrete symmetry for flavor physics and would 
give a large step to understand the origin of flavor. 

Let us consider SU{5) as a minimal candidate for unifled gauge group. For matter 
Yukawa couplings, we introduce as usual the three-generation fermions, ipi{10), 

{i = 1,2,3) and two scalars, hu{5) and hd{5*). Needless to say, one needs some 
different types of scalar flelds to break the unifled gauge symmetry, but they are generally 
irrelevant to the Yukawa sector. We deflne the flavor S3 charges of matter and scalar fields 




(4.9) 
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so that the combinations ('?/^i,'^2), (Xi^Xs) and (z/i, 1^2) are S3 doublets and the others in 
the Is representation. To implement the Ss invariance, a doublet and a singlet scalars 
are also introduced, called and Hs, respectively. The total field content is listed in 
the table below. We assume that ^'3 is broken by the scalar VEVs; (i^o) = ((-f^i), 0) and 
(Hs), which are of 0{1) and can be the result of analyzing symmetry-invariant scalar 
potential without fine tuning. 



While the right-handed neutrinos are assigned to the S3 representations (z/i, 1/2) + z/3, 
that is not essential as long as the low-energy left-handed Majorana masses are concerned. 
It is because, in the seesaw mechanism, arbitrary rotations of right-handed neutrinos can 
be inserted while preserving left-handed Majorana mass matrix. In fact, the 83 fiavor 
charges (i.e. the fiavor basis) of i/i turn out to be fixed, once some other dynamics than 
S3 is incorporated. 

In addition to these, we have a C/(l) (or Zn) symmetry to produce natural hierarchies 
among matrix elements (and also to help to discriminate i/i). A key of the mechanism is 
the existence of a scalar field (p with a non-vanishing VEV, which is a bit smaller than 
the fundamental scale of the theory. The whole ^'3 representations and C/(l) quantum 
numbers are shown in the table {x >0 and 2 > y > for phenomenological requirements) . 

Let us first discuss the Majorana masses of right-handed neutrinos. They are given 
by the following form of S'3-invariant mass operator ^^{MRjijVj: 



where the mass parameters M and M' are generally free. They contain some compound 
factors such as suppressions by fiavor U{1) invariance, but such suppression factors are 
cancelled out via the seesaw mechanism and irrelevant for low-energy quantity. The orders 
of magnitudes of M and M' would be determined by experimental data. Theoretically 
this seems natural since Mr cannot be directly induced by hu,d-i nor come from Hd,s-i as 
the Majorana mass term violates lepton number symmetry. Otherwise one easily obtains 
Mr by assuming VEVs of 5'3-singlet scalar fields with lepton numbers. In any case, the 
most important point here is that a single 5*3 fiavor symmetry controls both Dirac and 
Majorana mass textures simultaneously. 



^1,2 ^3 Xl,3 X2 1^1,2 1^3 K hd Hp Hs (p 



SU{5) 

S3 
Uil) 



10 10 5* 5* 1 1 5 5* 1 1 1 
2121 2 111211 
4 2 0x + 2y-2 -2 2 -1 




(5.1) 
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We also find that the 6*3 and U{1) flavor invariance leads to the following Dirac mass 
textures Mu,d,e,u for quarks and leptons: 



M„ = AM K, M, = A^ A2 hd, (5.2 





/A"+2 A^+2\ 
Me = Mj, M, = A^+2 y (53) 

\ \y \y J 

where we have neglected coupling constants and defined the expansion parameter A as 
the ratio of {(p) to the fundamental scale. Our first prediction obtained from the above 
textures is that the VEV ratio of two SU{2)]y doublets should be large, that is, hu/hd ~ 
Tnt/rrib ~ 60. Secondly, we find the Majorana mass matrix Ml of left-handed neutrinos 
derived from the seesaw formula 

Here the parameter e has been defined as e = A^^~^^"'"^(M'/M). The induced matrix 
elements in the 1-1, 1-3 and 3-1 positions are negligibly small and have been dropped 
in the expression (j5.4j) . It is interesting to note that all the symbols "1" in the matrix 
()5.4|) are exactly 1 due to the seesaw mechanism, and consequently, the lower-right 2 x 
2 sub-matrix has a reduced determinant of 0{e), not 0{1). For a small e (~ A^~^), 
the matrix ()5.4|) automatically realizes the bi-large generation mixing with hierarchical 
neutrino spectrum. The model also predicts a small lepton mixing of 0{e) between the 
first and third generations. The planned improvements in the sensitivity to such a small 
angle are expected to reach (9(10~^) ^28^, and the above model will be testable in near 
future. On the other hand, the neutrinoless double beta decay [21] cannot be observed 
due to the negligible value of (Ml)ii. 

It is found in the above Dirac mass matrices that some distorted values are induced 
for light generations, while the coefficients of matrix elements could be appropriately 
chosen for reproducing the observed values of fermion masses and mixing angles. We 
checked that, for the present simple field content, the flavor charge assignment of 5*3 
and U{1) symmetries is uniquely determined for realizing the two conditions that (i) the 
right-handed neutrino masses come from the mass operator (j5.H) and (ii) the rank of 
2x2 sub-matrix of Ml is reduced via the seesaw mechanism without tuning of couplings. 
Therefore an improvement of the above toy model is made by extending it to include 
additional scalar fields and modifying the texture forms. For example, the equivalence of 
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the down and charged-lepton mass spectrum may be split by making use of Higgs fields in 
higher- dimensional representations. Another simple extension is to change the S3 group 
basis, e.g. to the real basis. This might be a possible amelioration because the texture 
analysis and its physical consequences depend on the group basis of flavor symmetry, as 
explained before. We leave these issues to future investigation. 

6 Analysis of scalar potentials 

We have so far discussed mass textures of fermions in the framework of S3 flavor symmetry, 
where the 5*3 invariance is violated only by scalar VEVs. In particular, the assumption 
that either of components in a doublet has a vanishing VEV leads to texture forms for 
quark and lepton mass matrices, which result in various types of phenomenologically viable 
flavor models. In this section, we present, as an existence proof, the analysis of scalar 
potentials which generate desired forms of symmetry-breaking VEVs. We also discuss 
model parameters and symmetry for obtaining such VEVs. It should be noted that, unlike 
in the case of continuous flavor symmetry, a VEV of doublet cannot be transformed to 
an arbitrary form by 5*3 rotations because of the discreteness of group operations (see 
Table P). Even if there is a case that such rotation is useful, it is physically meaningful 
only when other symmetries in the theory do not commute with 5*3. This is not necessarily 
satisfied, for example, in the presence of U{1) flavor symmetry. 

6.1 The general case 

We consider a single 5*3 doublet scalar H = {Hi, H2) and analyze the structure of its 
generic scalar potential. The potential analysis becomes more involved if there are some 
numbers of scalar flelds. However since we have shown that mass textures of quarks and 
leptons can be derived from a single doublet scalar, the analysis for one doublet would 
provide a step towards more practical cases with more than one doublets. Furthermore, 
when several scalar flelds have VEVs of enough separated scales, they can be indepen- 
dently analyzed if the most generic terms are taken into account in scalar potentials. 

The most general and renormalizable scalar potential for a doublet H = {Hi,H2) is 
given by 

V{Hi,H2) = fi\HlHi + HlH2) + X[iHlHiY + {HlH2f] 

+ [p^HiH2 + a{Hl + Hi) + (3{HlHl + HIhI) + h.c] 
+kHIHiHIH2 + [CH^Hl + r]{H\HlH2 + hIhIH^) + h.c] (6.1) 
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with respect to the S3 invariance in the complex basis. The minimization of the potential 
leads to the following two conditions: 

+ 2\H\hI + p^*hI + 3a*i/f + pHl + 2(3*H\H2 + kHiHIH2 

+2CHIhI^ + r]HlH2 + r]*{2HlH,Hl + //f (6.2) 

I2^H2 + 2\hIhI + p^*hI + 3a*i7f + ^Hl + 2(3*HiHI + kHIHiH2 

+2CHI^hI + r]HiHl + r]*{Hl^H^ + 2HIHIH2). (6.3) 

The minimum is ensured by examining whether the 2x2 matrix d^V/dHidHj = 1, 2) 
has positive eigenvalues at the extreme. Generally, the stationary conditions ()6.2|) and 
()6.3|1 are satisfied at various points in the field space. The origin Hi = H2 = is a 
solution of these two condition, but at this point, the 5*3 flavor symmetry is unbroken and 
the result is unrealistic. Parallel to the well-known electroweak symmetry breaking, one 
easily avoids this trivial solution by assuming a negative value of /i^, which in turn makes 
the origin being a locally maximum point and destabilized.^ A possible minimum we are 
interested in is that either of VEVs in a doublet scalar is vanishing, namely Hi ^ and 
H2 = 0, or vice versa. For H2 = 0, the stationary conditions ()6.2|) and ()6.3|) become 

H^Hi + 'ia*Hl^ + 2XHIhI = 0, (6.4) 
p'^Hi+ (3*hI^ + r]HlHf = 0. (6.5) 

It is found that the existence of non-trivial (at least local) minimum with Hi ^ requires 
a set of conditions for the coupling constants. For example, p = f3 = r] = 0oT /i^/p^ = 
Sa/Z? = 2X/ri, and so on. The former condition could easily be obtained if the scalar 
has some other quantum charges than 5*3. There exist various possibilities of coupling 
constants for stabilizing the scalar potential at non-trivial minimum. It is an intriguing 
task to find the conditions of couplings which are naturally realized with symmetries. 
Finally we comment on the VEV form Hi ^ 0, H2 ^ 0. Such a form of doublet VEVs is 
needed to achieve the texture form discussed in Section 14.41 If the coupling constants in 
the above scalar potential have the same order of magnitude (in the unit of fundamental 
scale), two VEVs Hi and H2 are on a similar order and generally have different values. 
In this case, it is difficult to write down the generic solution to ()6.2p and ()6.3p for the 
potential minimum. We numerically investigated that there indeed exists the minimum 
of the potential 1)6.11) . As an example, a set of natural values of coupling constants, 
p = p = a = (3 = m and \ = K, = (^ = ri = l, leads to the stationary point Hi = 0.61m 

^Exactly speaking, if /i^ < \p^\, the origin of field space is destabilized, but only along with limited 
directions. 



dV 
dHl 



dV 
dHl 
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and H2 = —1.62m. It can easily be checked that this point corresponds to a potential 
minimum by analyzing the second order derivatives of the potential. It is also interesting 
to explore the region of couphng constants for which such a form of doublet VEVs appears. 



6.2 Supersymmetric case 

The analysis is similarly performed in supersymmetric theory. For an S'3-doublet superfield 
H = {Hi,H2), the symmetry- invariant superpotential W is given by 

W = mH,H2 + l{Hf+Hl) + ^H',H^+xH,H2{Hl + Hl) + ^{H! + Hl) + ^HlHl (6.6) 
6 2 60 

where m, y, ■ ■ ■ , zi, Z2 are the coupling constants and we have included the most general 
operators with mass dimensions up to seven. In this subsection, we assume that super- 
symmetry is unbroken, that is, supersymmetry-breaking soft terms do not fix the breaking 
scale of flavor symmetry. The scalar potential is then given by 

V{Hi, H2) = mH, + yHl + wHIH^ + x{4HiHl + Hf) + z^Hl + Z2HIHI ' 

+ \mH2 + yHl + wH^Hl + x{4HfH2 + H^) + ziHl + Z2HIHI? (<6.1) 

Comparing to the general case ()6.1|) . we have the relations from supersymmetry: 

= |m|^, A = |?/|^, (3 = m*y, r] = m*w, p'^ = a = K = ( = 0. (6.8) 

We here focus on some simple cases obtained by introducing discrete symmetry on 
the superpotential. The vacuum is where one of the VEVs of doublet is vanishing, e.g. 
H = {Hi, 0). Let us consider a symmetry of which the superfield H has a charge +1. 
In this case, it permits only the y, z\ and Z2 terms in the superpotential ()6.6p . As in the 
general case without supersymmetry, the origin i^i = = is a trivial solution without 
flavor symmetry breaking. We here assume for simplicity that the origin is destabilized if 
supersymmetry-breaking (and ^s-breaking) soft scalar terms are included. We then find a 
potential minimum where the scalars develop their VEVs of the form H = {{—y/ziY^'^, O) . 
At this vacuum, a vanishing VEV can give rise to mass textures (zeros) for quarks and 
leptons as explained before. Another simple example is a discrete R symmetry Z-^ji under 
which the superfield H has a charge +1. In this case, only the m and x terms are allowed 
in the superpotential (j6.6|) . Similarly assuming that the origin does not correspond to 
the minimum, we find a vacuum at which the doublet has the one-sided form of VEVs 
H = {{—m/xY^^, 0). It may be interesting to note that the ^3(_r) symmetry is suitable for 
discussing trilinear Yukawa terms and also for suppressing non-renormalizable operators 
involving more than one 5*3 doublets. We finally comment that the analysis becomes more 
involved if higher- dimensional operators in Kahler terms were included. 
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7 Summary and discussions 



Discrete flavor symmetry is a powerful instrument in classifying and constructing mass 
matrix forms of quarks and leptons. It is greatly anticipated that such discrete symmetry 
is practically inherited from continuous symmetries in more fundamental theory in high- 
energy regime. In this paper, we focus on the S3 group as a promising candidate of 
such symmetry. The 5*3 is the smallest non-abelian discrete symmetry and also has a 
simple geometrical interpretation. After discussing some fundamental issues of the 5*3 
group (representations, tensor products and group basis dependence), we have presented 
the general forms of 3 x 3 mass matrices which are derived from various types of S3 
theories. Based on that prescription, we have performed the construction of particular 
ansatze of mass textures which have been often discussed in the literature. That includes 
asymmetrical matrices in generation space, which would be appropriate to discussing the 
leptonic flavor mixing. 

Furthermore, discrete flavor symmetry can control the structure of mass matrices (tex- 
ture) in dynamical ways. In particular, we have shown that vanishing matrix elements 
(texture zeros) are dynamically realized in the vacuum of scalar potential. It is however 
noted that texture zeros themselves do not give any explanation for fermion mass hier- 
archies. The observed values of masses and mixing angles are obtained in our scheme by 
introducing a U{1) or symmetry. A non-trivial issue here is whether abelian charges 
can be assigned so that they are compatible with non-abelian flavor symmetry. We have 
described typical examples of charge assignments which not only produce mass hierarchy 
among the generations but also suppress higher- dimensional operators which tend to dis- 
turb mass hierarchies in the first-order estimation. Unlike previous approaches with the 
5*3 group, we do not assume hierarchical coupling constants nor sequential breaking of 
flavor symmetry. In realistic flavor models for quarks and leptons, it is important that 
several different sectors are simultaneously controlled by flavor symmetry. As an illus- 
trative example, a grand unified model has been constructed with a single S3 symmetry. 
Finally, we have also analyzed the scalar potentials for S'3-doublet scalars, whose form of 
VEV is a key ingredient of our approach. 

In this paper (especially below Section H)), we have focused on the models with a single 
S3 symmetry. It may be an interesting task to explore other types of flavor extensions, 
e.g. S3L X Ssji and other discrete symmetries larger than 5*3. They would generally lead 
to different types of mass textures and physical consequences such as flavor-violating 
processes and their characteristic experimental signatures. More detailed study, including 
these issues, is left to future investigations. 



33 



Acknowledgments 

We would like to thank M. Tanimoto for helpful discussion. This work is supported by 
scientific grants from the Ministry of Education (No. 16028214, 16540258, 17740150) and 
by grant-in-aid for the scientific research on priority area "Progress in elementary parti- 
cle physics of the 21st century through discoveries of Higgs boson and supersymmetry" 
(No. 16081209). 



34 



References 

[1] H. Pritzsch, Phys. Lett. B73 (1978) 317. 

[2] H. Georgi and C. Jarlskog, Phys. Lett. B86 (1979) 297. 

[3] G.F. Giudice, Mod. Phys. Lett. A7 (1992) 2429. 

[4] P. Ramond, R.G. Roberts and G.G. Ross, Nucl. Phys. B406 (1993) 19. 

[5] P.H. Prampton, S.L. Glashow and D. Marfatia, Phys. Lett. B536 (2002) 79. 

[6] Y. Pukuda et al, Phys. Lett. B433 (1998) 9; zMd. B436 (1998) 33; Phys. Rev. Lett. 
81 (1998) 1562; ibid. 82 (1999) 2644; ibid. 85 (2000) 3999. 

[7] S. Pukuda et al, Phys. Rev. Lett. 86 (2001) 5651; ibid. 86 (2001) 5656; Phys. Lett. 
B539 (2002) 179; Q.R. Ahmad et al, Phys. Rev. Lett. 89 (2002) 011301; ibid. 89 
(2002) 011302; K. Eguchi et ai, Phys. Rev. Lett. 90 (2003) 021802; T. Araki et al., 
Phys. Rev. Lett. 94 (2005) 081801. 

[8] S.P. King, Phys. Lett. B439 (1998) 350; JHEP 0209 (2002) Oil; S. Davidson and 
S.P. King, Phys. Lett. B445 (1998) 191. 

[9] A. Santamaria and J.W.P. Valle, Phys. Lett. B195 (1987) 423; R. Hempfling, 
Nucl. Phys. B478 (1996) 3; P.M. Borzumati, Y. Grossman, E. Nardi and Y. Nir, 
Phys. Lett. B384 (1996) 123; M. Drees, S. Pakvasa, X. Tata and T. ter Veldhuis, 
Phys. Rev. D57 (1998) 5335; E.J. Chun, S.K. Kang, C.W. Kim and U.W. Lee, 
Nucl. Phys. B544 (1999) 89; V. Bednyakov, A. Paessler and S. Kovalenko, Phys. Lett. 
B442 (1998) 203; B. Mukhopadhyaya, S. Roy and P. Vissani, Phys. Lett. B443 
(1998) 191; O.C.W. Kong, Mod. Phys. Lett. A14 (1999) 903; J. Perrandis, Phys. Rev. 
D60 (1999) 095012; D.E. Kaplan and A.E. Nelson, JHEP 0001 (2000) 033; A.S. Jo- 
shipura and S.K. Vempati, Phys. Rev. D60 (1999) 111303; A. Abada and M. Losada, 
Nucl. Phys. B585 (2000) 45; M. Hirsch, M.A. Diaz, W. Porod, J.C. Romao and 
J.W.P. Valle, Phys. Rev. D62 (2000) 113008 [Erratum-ibid. D65 (2002) 119901]; 
S. Davidson and M. Losada, Phys. Rev. D65 (2002) 075025. 

[10] K.S. Babu and S.M. Barr, Phys. Lett. B381 (1996) 202; J. Sato and T. Yanagida, 
Phys. Lett. B430 (1998) 127; C.H. Albright, K.S. Babu and S.M. Barr, 
Phys. Rev. Lett. 81 (1998) 1167; J.K. Elwood, N. Irges and P. Ramond, 
Phys. Rev. Lett. 81 (1998) 5064; Y. Nomura and T. Yanagida, Phys. Rev. D59 



35 



(1999) 017303; N. Haba, Phys. Rev. D59 (1999) 035011; G. Altarelli and F. Fer- 
uglio, Phys. Lett. B451 (1999) 388; W. BuchmuUer and T. Yanagida, Phys. Lett. 
B445 (1999) 399; Z. Berezhiani and A. Rossi, JHEP 9903 (1999) 002; M. Bando and 
T. Kugo, Prog. Theor. Phys. 101 (1999) 1313; S. Lola and G. G. Ross, Nucl. Phys. 
B553 (1999) 81; Y. Nir and Y. Shadmi, JHEP 9905 (1999) 023; K. Yoshioka, 
Mod. Phys. Lett. A15 (2000) 29; RH. Frampton and A. Rasin, Phys. Lett. B478 

(2000) 424; M. Bando, T. Kugo and K. Yoshioka, Prog. Theor. Phys. 104 (2000) 211; 
Phys. Lett. B483 (2000) 163; J.M. Mira, E. Nardi, D.A. Restrepo and J.W.F. Valle, 
Phys. Lett. B492 (2000) 81; M. Bando et al, Phys. Rev. D63 (2001) 113017; N. Haba 
and H. Murayama, Phys. Rev. D63 (2001) 053010; M.S. Bcrgcr and K. Siyeon, 
Phys. Rev. D63 (2001) 057302; A. Kageyama, M. Tanimoto and K. Yoshioka, 
Phys. Lett. B512 (2001) 349; K.S. Babu and S. M. Barr, Phys. Lett. B525 (2002) 
289; X.J. Bi and Y.B. Dai, Eur. Phys. J. C27 (2003) 43; K.S. Babu, I. Gogoladze 
and K. Wang, Nucl. Phys. B660 (2003) 322; R. Kitano and T. j. Li, Phys. Rev. D67 
(2003) 116004; T. Asaka, Phys. Lett. B562 (2003) 291; K.S. Babu, T. Enkhbat and 
I. Gogoladze, Nucl. Phys. B678 (2004) 233. 

[11] J. Sato, K. Tobe and T. Yanagida, Phys. Lett. B498 (2001) 189; J. Sato and K. Tobe, 
Phys. Rev. D63 (2001) 116010. 

[12] A.Y. Smirnov, Phys. Rev. D48 (1993) 3264; Nucl. Phys. B466 (1996) 25; M. Tan- 
imoto, Phys. Lett. B345 (1995) 477; M. Bando, T. Kugo and K. Yoshioka, 
Phys. Rev. Lett. 80 (1998) 3004; M. Jezabek and Y. Sumino, Phys. Lett. B440 
(1998) 327; M. Bando and K. Yoshioka, Prog. Theor. Phys. 100 (1998) 1239; D. Fal- 
cone, Phys. Rev. D61 (2000) 097302; E.K. Akhmedov, G.C. Branco and M.N. Rebelo, 
Phys. Lett. B478 (2000) 215; T.K. Kuo, G.H. Wu and S.W. Mansour, Phys. Rev. 
D61 (2000) 111301; A. Datta, F.S. Ling and P. Ramond, Nucl. Phys. B671 (2003) 
383; M. Honda, S. Kaneko and M. Tanimoto, Phys. Lett. B593 (2004) 165; R. Der- 
misek, Phys. Rev. D70 (2004) 073016; M. Lindner, M.A. Schmidt and A.Y. Smirnov, 
JHEP 0507 (2005) 048. 

[13] E. Derman and H.S. Tsao, Phys. Rev. D20 (1979) 1207; Y. Yamanaka, H. Sug- 
awara and S. Pakvasa, Phys. Rev. D25 (1982) 1895 [Erratum-ibid. D29 (1984) 
2135]; G. Ecker, Z. Phys. C24 (1984) 353; D. Chang, W. Y. Keung and G. Sen- 
janovic, Phys. Rev. D42 (1990) 1599; D.B. Kaplan and M. Schmaltz, Phys. Rev. 
D49 (1994) 3741; P.H. Frampton and T.W. Kephart, Int. J. Mod. Phys. AlO (1995) 
4689; M. Schmaltz, Phys. Rev. D52 (1995) 1643; P.H. Frampton and O.C.W. Kong, 
Phys. Rev. Lett. 77 (1996) 1699; A. Aranda, CD. Carone and R.F. Lebed, 



36 



Phys. Lett. B474 (2000) 170; E. Ma and G. Rajasekaran, Phys. Rev. D64 (2001) 
113012; K.S. Babu, E. Ma and J.W.F. Valle, Phys. Lett. B552 (2003) 207; E. Ma, 
Mod. Phys. Lett. A17 (2002) 2361; W. Grimus and L. Lavoura, Phys. Lett. B572 
(2003) 189; C.I. Low and R.R. Volkas, Phys. Rev. D68 (2003) 033007; M. Hirsch, 
J.C. Romao, S. Skadhauge, J.W.F. Valle and A. Villanova del Moral, Phys. Rev. 
D69 (2004) 093006; W. Grimus, A.S. Josliipura, S. Kaneko, L. Lavoura and M. Tan- 
imoto, JHEP 0407 (2004) 078; M. Frigerio, S. Kaneko, E. Ma and M. Tanimoto, 
Phys. Rev. D71 (2005) 011901; K.S. Babu and J. Kubo, Phys. Rev. D71 (2005) 
056006; G. Altarelli and F. Feruglio, Nud. Phys. B720 (2005) 64; A. Zee, Phys. Lett. 
B630 (2005) 58. 

[14] H. Harari, H. Haut and J. Weyers, Phys. Lett. B78 (1978) 459; Y. Koide, Phys. Rev. 
D28 (1983) 252; %bzd. D39 (1989) 1391; R Kaus and S. Meshkov, Mod. Phys. Lett. 
A3 (1988) 1251 [Erratum-ibid. A4 (1989) 603]; L. Lavoura, Phys. Lett. B228 
(1989) 245; M. Tanimoto, Phys. Rev. D41 (1990) 1586; G.C. Branco, J.I. Silva- 
Marcos and M.N. Rebelo, Phys. Lett. B237 (1990) 446; H. Fritzsch and J. Plankl, 
Phys. Lett. B237 (1990) 451; H. Fritzsch and Z.Z. Xing, Phys. Lett. B372 (1996) 
265; M. Fukugita, M. Tanimoto and T. Yanagida, Phys. Rev. D57 (1998) 4429; 
R.N. Mohapatra and S. Nussinov, Phys. Lett. B441 (1998) 299; P.F. Harrison 
and W.G. Scott, Phys. Lett. B557 (2003) 76; F. Caravaglios and S. Morisi, 
|arXiv:hep-ph/0503234 

[15] S. Pakvasa and H. Sugawara, Phys. Lett. B73 (1978) 61; E. Derman, Phys. Rev. 
D19 (1979) 317; L.J. Hall and H. Murayama, Phys. Rev. Lett. 75 (1995) 
3985; CD. Carone, L.J. Hall and H. Murayama, Phys. Rev. D53 (1996) 6282; 
R.N. Mohapatra, A. Perez-Lorenzana and C.A. de Sousa Pires, Phys. Lett. B474 
(2000) 355; J. Kubo, A. Mondragon, M. Mondragon and E. Rodriguez- Jauregui, 
Prog. Theor. Phys. 109 (2003) 795; T. Kobayashi, J. Kubo and H. Terao, Phys. Lett. 
B568 (2003) 83; J. Kubo, H. Okada and F. Sakamaki, Phys. Rev. D70 (2004) 036007; 
T. Araki, J. Kubo and E.A. Paschos, Eur. Phys. J. C45 (2006) 465. 

[16] D. Wyler, Phys. Rev. D19 (1979) 330; E. Ma, Phys. Rev. D43 (1991) 2761; 
N.G. Deshpande, M. Gupta and P.B. Pal, Phys. Rev. D45 (1992) 953; R. Dermisek 
and S. Raby, Phys. Rev. D62 (2000) 015007; S.L. Chen, M. Frigerio and E. Ma, 
Phys. Rev. D70 (2004) 073008 [Erratum-ibid. D70 (2004) 079905]; W. Grimus and 
L. Lavoura, JHEP 0508 (2005) 013. 

[17] S. Dimopoulos and H. Georgi, NucL Phys. B193 (1981) 150; J.R. Elhs and 
D.V. Nanopoulos, Phys. Lett. BllO (1982) 44; J.F. Donoghue, H.P. Nilles and 



37 



D. Wyler, Phys. Lett. B128 (1983) 55; L.J. Hall, V.A. Kostelecky and S. Raby, 
Nucl. Phys. B267 (1986) 415; F. Gabbiani and A. Masiero, Nucl. Phys. B322 (1989) 
235; J.S. Hagelin, S. Kelley and T. Tanaka, Nucl. Phys. B415 (1994) 293; F. Gab- 
biani, E. Gabrielli, A. Masiero and L. Silvestrini, Nucl. Phys. B477 (1996) 321. 

[18] A.H. Chamscddine, R. Arnowitt and R Nath, Phys. Rev. Lett. 49 (1982) 970; R. Bar- 
bieri, S. Fcrrara and C.A. Savoy, Phys. Lett. B119 (1982) 343; L.J. Hall, J. Lykken 
and S. Weinberg, Phys. Rev. D27 (1983) 2359; N. Ohta, Prog. Theor. Phys. 70 
(1983) 542; H.P. Nilles, Phys. Rept. 110 (1984) 1. 

[19] CD. Froggatt and H.B. Nielsen, Nucl Phys. B147 (1979) 277. 

[20] M. Leurer, Y. Nir and N. Seiberg, Nucl. Phys. B420 (1994) 468; L.E. Ibancz and 
G.G. Ross, Phys. Lett. B332 (1994) 100; E. Dudas, S. Pokorski and C.A. Savoy, 
Phys. Lett. B356 (1995) 45; P. Binetruy, S. Lavignac and P. Ramond, Nucl. Phys. 
B477 (1996) 353; R.N. Mohapatra and A. Riotto, Phys. Rev. D55 (1997) 4262; 
K. Choi, E.J. Chun and H.D. Kim, Phys. Lett. B394 (1997) 89; A.E. Nelson and 
D. Wright, Phys. Rev. D56 (1997) 1598; Y. Grossman, Y. Nir and Y. Shadmi, JHEP 
9810 (1998) 007; S.F. King, Nucl Phys. B576 (2000) 85; Q. Shafi and Z. Tavartk- 
iladze, Phys. Lett. B482 (2000) 145; N. Maekawa, Prog. Theor. Phys. 106 (2001) 
401; N. Maekawa and T. Yamashita, Prog. Theor. Phys. 107 (2002) 1201; M. Kak- 
izaki and M. Yamaguchi, JHEP 0206 (2002) 032; H.K. Dreiner, H. Murayama and 
M. Thormeier, Nucl. Phys. B729 (2005) 278; P.H. Chankowski, K. Kowalska, S. Lav- 
ignac and S. Pokorski, Phys. Rev. D71 (2005) 055004. 

[21] D.s. Du and Z.z. Xing, Phys. Rev. D48 (1993) 2349; H. Fritzsch and Z.z. Xing, 
Phys. Lett. B353 (1995) 114; P.S. Gill and M. Gupta, Phys. Rev. D56 (1997) 
3143; J.L. Chkareuh and CD. Froggatt, Phys. Lett. B450 (1999) 158; H. Nishiura, 
K. Matsuda and T. Fukuyama, Phys. Rev. D60 (1999) 013006; D. Falcone and 
F. Tramontano, Phys. Rev. D63 (2001) 073007; R. Rosenfeld and J.L. Rosner, 
Phys. Lett. B516 (2001) 408; W. BuchmuUer and D. Wyler, Phys. Lett. B521 (2001) 
291; J.L. Chkareuh, CD. Froggatt and H.B. Nielsen, Nucl. Phys. B626 (2002) 307; 
M. Bando and M. Obara, Prog. Theor. Phys. 109 (2003) 995. 

[22] G.C. Branco and J.L Silva-Marcos, Phys. Lett. B331 (1994) 390; T.K. Kuo, 
S.W. Mansour and G.H. Wu, Phys. Rev. D60 (1999) 093004; G.C. Branco, 
D. Emmanuel-Costa and R. Gonzalez Felipe, Phys. Lett. B477 (2000) 147. 

[23] N. Uekusa, A. Watanabe and K. Yoshioka, Phys. Rev. D71 (2005) 094024. 



38 



[24] G. Altarelli, F. Feruglio and 1. Masina, Phys. Lett. B472 (2000) 382. 

[25] R. Kuchimanchi and R.N. Mohapatra, Phys. Rev. D66 (2002) 051301; T. Endoh, 
S. Kaneko, S.K. Kang, T. Morozumi and M. Tanimoto, Phys. Rev. Lett. 89 (2002) 
231601; M. Raidal and A. Strumia, Phys. Lett. B553 (2003) 72; S.F. King, Phys. Rev. 
D67 (2003) 113010; S. Raby, Phys. Lett. B561 (2003) 119; B. Dutta and R.N. Mo- 
hapatra, Phys. Rev. D68 (2003) 056006; V. Barger, D.A. Dicus, H.J. He and T.j. Li, 
Phys. Lett. B583 (2004) 173; W.l. Guo and Z.z. Xing, Phys. Lett. B583 (2004) 163; 
W. Rodejohann, Eur. Phys. J. C32 (2004) 235; A. Ibarra and G.G. Ross, Phys. Lett. 
B591 (2004) 285; S.h. Chang, S.K. Kang and K. Siyeon, Phys. Lett. B597 (2004) 
78. 

[26] T. Yanagida, in Proceedings of the Workshop on Unihed Theories and Baryon Num- 
ber in the Universe, eds. O. Sawada and A. Sugamoto (KEK report 79-18, 1979); 
M. Gell-Mann, P. Ramond and R. Slansky, in Supergravity, eds. P. van Nieuwen- 
huizen and D.Z. Freedman (North Holland, Amsterdam, 1979). 

[27] P.H. Frampton, S.L. Glashow and T. Yanagida, Phys. Lett. B548 (2002) 119. 

[28] A. Cervera, A. Donini, M.B. Gavela, J.J. Gomez Cadenas, P. Hernandez, O. Mena 
and S. Rigolin, Nud. Phys. B579 (2000) 17; P. Huber, M. Lindner and W. Winter, 
Nud. Phys. B645 (2002) 3; P. Huber, M. Lindner, T. Schwetz and W. Winter, 
Nud. Phys. B665 (2003) 487. 

[29] For a recent review, see, S.R. Elliott and P. Vogel, Ann. Rev. Nud. Part. Sci. 52 
(2002) 115 [arXiv: |hg^^/ 0202264|. 



39 



